Design and Execution of make-like, distributed Analyses based on Spotify's Pipelining Package Luigi

摘要

In high-energy particle physics, workflow management systems are primarilyused as tailored solutions in dedicated areas such as Monte Carlo production.However, physicists performing data analyses are usually required to steertheir individual workflows manually which is time-consuming and often leads toundocumented relations between particular workloads. We present a genericanalysis design pattern that copes with the sophisticated demands of end-to-endHEP analyses and provides a make-like execution system. It is based on theopen-source pipelining package Luigi which was developed at Spotify and enablesthe definition of arbitrary workloads, so-called Tasks, and the dependenciesbetween them in a lightweight and scalable structure. Further features aremulti-user support, automated dependency resolution and error handling, centralscheduling, and status visualization in the web. In addition to alreadybuilt-in features for remote jobs and file systems like Hadoop and HDFS, weadded support for WLCG infrastructure such as LSF and CREAM job submission, aswell as remote file access through the Grid File Access Library. Furthermore,we implemented automated resubmission functionality, software sandboxing, and acommand line interface with auto-completion for a convenient workingenvironment. For the implementation of a $t\bar{t}H$ cross section measurement,we created a generic Python interface that provides programmatic access to allexternal information such as datasets, physics processes, statistical models,and additional files and values. In summary, the setup enables the execution ofthe entire analysis in a parallelized and distributed fashion with a singlecommand.